Evidence for Model-based Computations in the Human Amygdala during Pavlovian Conditioning

Contemporary computational accounts of instrumental conditioning have emphasized a role for a model-based system in which values are computed with reference to a rich model of the structure of the world, and a model-free system in which values are updated without encoding such structure. Much less studied is the possibility of a similar distinction operating at the level of Pavlovian conditioning. In the present study, we scanned human participants while they participated in a Pavlovian conditioning task with a simple structure while measuring activity in the human amygdala using a high-resolution fMRI protocol. After fitting a model-based algorithm and a variety of model-free algorithms to the fMRI data, we found evidence for the superiority of a model-based algorithm in accounting for activity in the amygdala compared to the model-free counterparts. These findings support an important role for model-based algorithms in describing the processes underpinning Pavlovian conditioning, as well as providing evidence of a role for the human amygdala in model-based inference

Stimulus-Dependent Adjustment of Reward Prediction Error in the Midbrain

Previous reports have described that neural activities in midbrain dopamine areas are sensitive to unexpected reward delivery and omission. These activities are correlated with reward prediction error in reinforcement learning models, the difference between predicted reward values and the obtained reward outcome. These findings suggest that the reward prediction error signal in the brain updates reward prediction through stimulus–reward experiences. It remains unknown, however, how sensory processing of reward-predicting stimuli contributes to the computation of reward prediction error. To elucidate this issue, we examined the relation between stimulus discriminability of the reward-predicting stimuli and the reward prediction error signal in the brain using functional magnetic resonance imaging (fMRI). Before main experiments, subjects learned an association between the orientation of a perceptually salient (high-contrast) Gabor patch and a juice reward. The subjects were then presented with lower-contrast Gabor patch stimuli to predict a reward. We calculated the correlation between fMRI signals and reward prediction error in two reinforcement learning models: a model including the modulation of reward prediction by stimulus discriminability and a model excluding this modulation. Results showed that fMRI signals in the midbrain are more highly correlated with reward prediction error in the model that includes stimulus discriminability than in the model that excludes stimulus discriminability. No regions showed higher correlation with the model that excludes stimulus discriminability. Moreover, results show that the difference in correlation between the two models was significant from the first session of the experiment, suggesting that the reward computation in the midbrain was modulated based on stimulus discriminability before learning a new contingency between perceptually ambiguous stimuli and a reward. These results suggest that the human reward system can incorporate the level of the stimulus discriminability flexibly into reward computations by modulating previously acquired reward values for a typical stimulus

The Value of Success: Acquiring Gains, Avoiding Losses, and Simply Being Successful

A large network of spatially contiguous, yet anatomically distinct regions in medial frontal cortex is involved in reward processing. Although it is clear these regions play a role in critical aspects of reward-related learning and decision-making, the individual contributions of each component remains unclear. We explored dissociations in reward processing throughout several key regions in the reward system and aimed to clarify the nature of previously observed outcome-related activity in a portion of anterior medial orbitofrontal cortex (mOFC). Specifically, we tested whether activity in anterior mOFC was related to processing successful actions, such that this region would respond similarly to rewards with and without tangible benefits, or whether this region instead encoded only quantifiable outcome values (e.g., money). Participants performed a task where they encountered monetary gains and losses (and non-gains and non-losses) during fMRI scanning. Critically, in addition to the outcomes with monetary consequences, the task included trials that provided outcomes without tangible benefits (participants were simply told that they were correct or incorrect). We found that anterior mOFC responded to all successful outcomes regardless of whether they carried tangible benefits (monetary gains and non-losses) or not (controls). These results support the hypothesis that anterior mOFC processes rewards in terms of a common currency and is capable of providing reward-based signals for everything we value, whether it be primary or secondary rewards or simply a successful experience without objectively quantifiable benefits

Dopamine restores reward prediction errors in old age.

Senescence affects the ability to utilize information about the likelihood of rewards for optimal decision-making. Using functional magnetic resonance imaging in humans, we found that healthy older adults had an abnormal signature of expected value, resulting in an incomplete reward prediction error (RPE) signal in the nucleus accumbens, a brain region that receives rich input projections from substantia nigra/ventral tegmental area (SN/VTA) dopaminergic neurons. Structural connectivity between SN/VTA and striatum, measured by diffusion tensor imaging, was tightly coupled to inter-individual differences in the expression of this expected reward value signal. The dopamine precursor levodopa (L-DOPA) increased the task-based learning rate and task performance in some older adults to the level of young adults. This drug effect was linked to restoration of a canonical neural RPE. Our results identify a neurochemical signature underlying abnormal reward processing in older adults and indicate that this can be modulated by L-DOPA

Beliefs about the Minds of Others Influence How We Process Sensory Information

Attending where others gaze is one of the most fundamental mechanisms of social cognition. The present study is the first to examine the impact of the attribution of mind to others on gaze-guided attentional orienting and its ERP correlates. Using a paradigm in which attention was guided to a location by the gaze of a centrally presented face, we manipulated participants' beliefs about the gazer: gaze behavior was believed to result either from operations of a mind or from a machine. In Experiment 1, beliefs were manipulated by cue identity (human or robot), while in Experiment 2, cue identity (robot) remained identical across conditions and beliefs were manipulated solely via instruction, which was irrelevant to the task. ERP results and behavior showed that participants' attention was guided by gaze only when gaze was believed to be controlled by a human. Specifically, the P1 was more enhanced for validly, relative to invalidly, cued targets only when participants believed the gaze behavior was the result of a mind, rather than of a machine. This shows that sensory gain control can be influenced by higher-order (task-irrelevant) beliefs about the observed scene. We propose a new interdisciplinary model of social attention, which integrates ideas from cognitive and social neuroscience, as well as philosophy in order to provide a framework for understanding a crucial aspect of how humans' beliefs about the observed scene influence sensory processing

Ventromedial Prefrontal Cortex Activation Is Associated with Memory Formation for Predictable Rewards

During reinforcement learning, dopamine release shifts from the moment of reward consumption to the time point when the reward can be predicted. Previous studies provide consistent evidence that reward-predicting cues enhance long-term memory (LTM) formation of these items via dopaminergic projections to the ventral striatum. However, it is less clear whether memory for items that do not precede a reward but are directly associated with reward consumption is also facilitated. Here, we investigated this question in an fMRI paradigm in which LTM for reward-predicting and neutral cues was compared to LTM for items presented during consumption of reliably predictable as compared to less predictable rewards. We observed activation of the ventral striatum and enhanced memory formation during reward anticipation. During processing of less predictable as compared to reliably predictable rewards, the ventral striatum was activated as well, but items associated with less predictable outcomes were remembered worse than items associated with reliably predictable outcomes. Processing of reliably predictable rewards activated the ventromedial prefrontal cortex (vmPFC), and vmPFC BOLD responses were associated with successful memory formation of these items. Taken together, these findings show that consumption of reliably predictable rewards facilitates LTM formation and is associated with activation of the vmPFC

Functional MRI in Awake Unrestrained Dogs

Because of dogs' prolonged evolution with humans, many of the canine cognitive skills are thought to represent a selection of traits that make dogs particularly sensitive to human cues. But how does the dog mind actually work? To develop a methodology to answer this question, we trained two dogs to remain motionless for the duration required to collect quality fMRI images by using positive reinforcement without sedation or physical restraints. The task was designed to determine which brain circuits differentially respond to human hand signals denoting the presence or absence of a food reward. Head motion within trials was less than 1 mm. Consistent with prior reinforcement learning literature, we observed caudate activation in both dogs in response to the hand signal denoting reward versus no-reward

Attention-dependent modulation of cortical taste circuits revealed by granger causality with signal-dependent noise

We show, for the first time, that in cortical areas, for example the insular, orbitofrontal, and lateral prefrontal cortex, there is signal-dependent noise in the fMRI blood-oxygen level dependent (BOLD) time series, with the variance of the noise increasing approximately linearly with the square of the signal. Classical Granger causal models are based on autoregressive models with time invariant covariance structure, and thus do not take this signal-dependent noise into account. To address this limitation, here we describe a Granger causal model with signal-dependent noise, and a novel, likelihood ratio test for causal inferences. We apply this approach to the data from an fMRI study to investigate the source of the top-down attentional control of taste intensity and taste pleasantness processing. The Granger causality with signal-dependent noise analysis reveals effects not identified by classical Granger causal analysis. In particular, there is a top-down effect from the posterior lateral prefrontal cortex to the insular taste cortex during attention to intensity but not to pleasantness, and there is a top-down effect from the anterior and posterior lateral prefrontal cortex to the orbitofrontal cortex during attention to pleasantness but not to intensity. In addition, there is stronger forward effective connectivity from the insular taste cortex to the orbitofrontal cortex during attention to pleasantness than during attention to intensity. These findings indicate the importance of explicitly modeling signal-dependent noise in functional neuroimaging, and reveal some of the processes involved in a biased activation theory of selective attention

Lung adenocarcinoma originates from retrovirus infection of proliferating type 2 pneumocytes during pulmonary post-natal development or tissue repair

Jaagsiekte sheep retrovirus (JSRV) is a unique oncogenic virus with distinctive biological properties. JSRV is the only virus causing a naturally occurring lung cancer (ovine pulmonary adenocarcinoma, OPA) and possessing a major structural protein that functions as a dominant oncoprotein. Lung cancer is the major cause of death among cancer patients. OPA can be an extremely useful animal model in order to identify the cells originating lung adenocarcinoma and to study the early events of pulmonary carcinogenesis. In this study, we demonstrated that lung adenocarcinoma in sheep originates from infection and transformation of proliferating type 2 pneumocytes (termed here lung alveolar proliferating cells, LAPCs). We excluded that OPA originates from a bronchioalveolar stem cell, or from mature post-mitotic type 2 pneumocytes or from either proliferating or non-proliferating Clara cells. We show that young animals possess abundant LAPCs and are highly susceptible to JSRV infection and transformation. On the contrary, healthy adult sheep, which are normally resistant to experimental OPA induction, exhibit a relatively low number of LAPCs and are resistant to JSRV infection of the respiratory epithelium. Importantly, induction of lung injury increased dramatically the number of LAPCs in adult sheep and rendered these animals fully susceptible to JSRV infection and transformation. Furthermore, we show that JSRV preferentially infects actively dividing cell in vitro. Overall, our study provides unique insights into pulmonary biology and carcinogenesis and suggests that JSRV and its host have reached an evolutionary equilibrium in which productive infection (and transformation) can occur only in cells that are scarce for most of the lifespan of the sheep. Our data also indicate that, at least in this model, inflammation can predispose to retroviral infection and cancer

Neural Correlates of Appetite and Hunger-Related Evaluative Judgments

How much we desire a meal depends on both the constituent foods and how hungry we are, though not every meal becomes more desirable with increasing hunger. The brain therefore needs to be able to integrate hunger and meal properties to compute the correct incentive value of a meal. The present study investigated the functional role of the amygdala and the orbitofrontal cortex in mediating hunger and dish attractiveness. Furthermore, it explored neural responses to dish descriptions particularly susceptible to value-increase following fasting. We instructed participants to rate how much they wanted food menu items while they were either hungry or sated, and compared the rating differences in these states. Our results point to the representation of food value in the amygdala, and to an integration of attractiveness with hunger level in the orbitofrontal cortex. Dishes particularly desirable during hunger activated the thalamus and the insula. Our results specify the functions of evaluative structures in the context of food attractiveness, and point to a complex neural representation of dish qualities which contribute to state-dependent value